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Abstract 

This  DoD  HPC  Challenge  project  is  a  crucial 
component  of  an  effort  to  develop  a  data-assimilative 
,  .  .j,  1/32°  global  ocean  nowcast/forecast  system,  which 

j  ffr  '  includes  the  associated  basic  research  and  exploratory 

f  development.  The  need  for  1/32°  resolution  (~3.5  km  at 

mid-latitudes)  has  been  demonstrated  through  extensive 
f;  research,  including  essential  contributions  from  our  FY97 
I'  Ifi-  FY98-00  DoD  HPC  Challenge  projects.  The  ocean 

J'v  model,  after  30  years  of  climatological  spinup,  was  run 
I  I  interannually  spanning  the  period  1979-present.  Data 

!  wj  assimilation  experiments  are  currently  underway. 

Transition  of  this  system  to  operational  use  at  the  Naval 
Oceanographic  Office  (NAVO)  is  planned  for  later  in 
2003. 

1.  Introduction 


This  DoD  HPC  Challenge  project  is  a  crucial 
component  of  an  effort  to  develop  a  data-assimilative 
1/32“  global  ocean  nowcast/forecast  system,  which 
includes  the  associated  basic  research  and  exploratory 
development.  The  need  for  1/32°  resolution  (—3.5  km  at 
mid-latitudes)  has  been  demonstrated  through  extensive 
research,  including  essential  contributions  from  our  FY97 
and  FY98-00  DoD  HPC  Challenge  projects.  These 
projects  also  helped  establish  1/16“  (-7  km  at  mid 
latitudes)  as  the  minimum  resolution  needed  for  a  fully 
eddy-resolving  global  ocean  model.  Major  improvement 
was  obtained  with  an  increase  to  1/32“  resolution,  but 
only  modest  additional  improvement  with  a  further 
increase  to  1/64“  resolution. 

Our  previous  DoD  HPC  Challenge  projects  were  a 
critical  component  of  the  effort  to  develop  a  1/16“  global 
ocean  nowcast/forecast  system.  That  system  has  been 
transitioned  to  the  Naval  Oceanographic  Office  (NAVO), 
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Stennis  Space  Center,  MS  where  it  has  been  running  in 
real  time  since  18  Oct  2000,  and  became  an  operational 
product  on  27  S^t  2001.  This  1/16“  system  includes 
assimilation  of  altimeter  sea  sirface  hei^t  (SSH)  data 
from  three  satellites  and  sea  surface  temperature  (SST) 
from  satellite  IR.  It  clearly  demonstrates  the  ability  to 
track  the  evolution  of  oceanic  eddies  and  the  meandering 
of  ocean  currents  and  fronts,  which  have  space  scales 
0(100  km).  At  least  30-day  forecast  skill  has  also  been 
demonstrated  globally.  Results  can  be  viewed  on  the  web 
at  http://wvyw.ocean.nrlssc.navv.mil/global  nlom 

The  full  resources  of  a  large  three-year  HPC 
Challenge  project  was  required  to  permit  the  transition 
later  in  2003  of  a  1/32°  global  ocean  nowcast/forecast 
system  to  NAVO.  That  includes  (a)  multi-decade  1/32“ 
global  ocean  simulations  which  are  needed  to  assess  the 
realism  and  dynamics  of  the  model  and  for  the  generation 
of  statistics  and  a  model  mean  SSH  needed  in  the  data 
assimilation,  (b)  ocean  nowcast  experiments  with  data 
assimilation  into  the  model  and  (c)  ocean  forecast 
experiments  initialized  from  the  experiments  with  data 
assimilation. 

Applications  for  this  nowcast/forecast  system  include 
assimilation  and  synthesis  of  global  satellite  surface  data; 
ocean  prediction;  optimum  track  ship  routing;  search  and 
rescue;  anti-submarine  warfare  and  surveillance;  tactical 
planning;  high  resolution  inputs  for  other  models  and 
shipbo^d  environmental  products;  environmental 
simulation  and  synthetic  environments;  observing  system 
simulations;  ocean  research;  pollution  and  tracer  tracking 
and  inputs  to  water  quality  assessment. 

The  NRL/NAVO  global  ocean  prediction  effort  is  a 
participant  in  the  multinational  Global  Ocean  Data 
Assimilation  Experiment  (GODAE)  wfrich  is  designed  to 
help  justify  a  permanent  global  ocean  observing  ^stem 
by  demonstrating  useful  real-time  global  ocean  products 
with  a  customer  base  (International  GODAE  Steering 
Team,  2000).  Our  effort  is  represented  on  the  U.S.  and 
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International  GODAE  Steering  Teams.  GODAE  is 
scheduled  for  2003-2007  with  pilot  programs  in  2000— 
2002. 

2.  Problem  and  Methodology 

Global  ocean  modeling  and  prediction  is  one  of  the 
original  Grand  Challenge  problems  that,  by  definition, 
require  a  Tflop/s  (sustained)  performance.  NRL’s 
approach  to  this  problem  seeks  to  reduce  the  cost  via  a 
careful  choice  of  algorithms  and  the  use  of  a  Lagrangian 
vertical  coordinate.  Because  the  NRL  Layered  Ocean 
Model  (NLOM)  has  a  factor  of  1  Os-1 00s  advantage  in 
computer  time  requirements  over  other  global  and  basin- 
scale  ocean  models,  we  can  “solve”  this  problem  on  a 
computer  sustaining  0(100)  Gflop/s.  The  NRL  modeling 
strategy  is  discussed  in  the  Hurlburt  and  Wallcraft  (2000) 
case  study  for  the  Smithsonian,  which  uses  the  case 
studies  as  one  means  of  developing  a  history  of 
information  technology  and  its  benefits  to  society.  NLOM 
(Wallcraft,  1991;  Wallcraft  and  Moore,  1997)  is  written  in 
the  tiled  data  parallel  programming  style.  It  is  scalable 
using:  (a)  autotasking,  or  (b)  data  parallel  Fortran,  or  (c) 
SHMEM  library  calls,  or  (d)  Message  Passing  Interface 
(MPI)  library  calls  and  thus  is  portable  to  a  wide  variety 
of  computing  platforms. 

A  critical  issue  in  forecast  system  design  is 
determining  the  resolution  required.  Ocean  models 
require  finer  resolution  and  more  computer  time  than 
atmospheric  models  in  part  because  the  space  scales  for 
variability  due  to  flow  instabilities  (oceanic  mesoscale 
eddies  vs.  atmospheric  highs  and  lows)  are  about  20-30 
times  smaller  than  found  in  the  atmosphere.  We  need  to 
resolve  the  oceanic  eddy  space  scale  very  well  because 
(1)  it  is  relevant  for  most  of  the  Navy  applications  listed 
earlier,  (2)  these  models  need  to  provide  high  resolution 
boimdaiy  conditions  for  even  higher  resolution  coastal 
models,  (3)  upper  ocean — ^topographic  coupling  via  flow 
instabilities  has  a  major  impact  on  the  pathways  of  many 
upper  ocean  currents  (including  mean  pathways)  and  very 
fine  resolution  of  the  flow  instabilities  is  required  to  get 
sufficient  coupling  (Hurlburt,  et.  al.,  1996;  Hurlburt  and 
Metzger,  1998;  Hogan  and  Hurlburt,  2000;  Tilburg  et.  al., 
2001),  (4)  very  fine  resolution  is  required  to  obtain  (a) 
inertial  jets  and  sharp  oceanic  fronts  which  span  major 
ocean  basins  as  observed  (Hurlburt,  et  al.,  1996)  and  (b) 
the  associated  nonlinear  recirculation  gyres  which  affect 
the  shape  of  large-scale  ocean  gyres  (Hurlburt  and  Hogan, 
2000),  (5)  it  is  necessary  to  resolve  small  islands  and 
narrow  passages  which  affect  current  pathways  and 
current  transports  in  many  regions  (e.g.,  Metzger  and 
Hurlburt,  2001),  (6)  in  data  assimilative  mode  we  do  not 
want  the  ocean  model  to  “fight”  the  data  because  the 
natural  behavior  of  the  ocean  model  is  inconsistent  with 


the  observations  (Hurlburt,  et  al.,  2000),  and  (7)  a  very 
high  horizontal  resolution  model  is  needed  to  help  get  an 
accurate  mean  sea  surface  height  field  to  add  to  the 
deviations  obtained  from  satellite  altimetry  (observations 
alone  do  not  provide  sufficient  resolution  to  do  this). 

The  need  for  1/32°  resolution  (~4  km)  has  been 
demonstrated  through  extensive  research,  including 
essential  contributions  from  our  FY97  and  FY98-00  DoD 
HPC  Challenge  projects  as  well  as  non-challenge  HPC 
usage  (e.g.,  the  publications  cited  above).  These  projects 
also  helped  establish  1/16°  (~7  km  at  mid  latitudes)  as  the 
minimum  resolution  needed  for  a  fully  eddy-resolving 
global  ocean  prediction  system.  Major  improvement  in 
model  simulation  skill  was  obtained  with  an  increase  to 
1/32°  resolution,  but  only  modest  additional  improvement 
with  a  fiarther  increase  to  1/64°  resolution  (Hogan  and 
Hurlburt,  2000;  Hurlburt  and  Hogan,  2000). 

The  1/32°  global  ocean  prediction  system  represents 
a  doubling  of  the  resolution  and  an  8-fold  increase  in 
computer  time  requirements  over  the  1/16°  global  system 
transitioned  to  NAVO  in  2000.  Therefore,  the  full 
resources  of  a  large  three-year  HPC  Challenge  project 
were  required  to  permit  transition  of  a  1/32°  global 
system  to  NAVO  in  2003.  In  the  first  step  climatological 
atmospheric  forcing  was  used  to  extend  the  equilibrated 
1/16°  global  model  at  1/32°  resolution  for  30  model  years 
until  it  was  re-equilibrated,  a  step  that  was  completed  in 
late  2001.  In  addition  to  re-equilibration,  the 
climatological  simulation  was  used  to  fix  problems  that 
arose,  optimize  model  parameters,  assess  the  impact  of 
the  resolution  increase  on  model  realism  and  dynamics 
and  to  perform  climatological  model-data  comparisons. 

The  climatological  simulation  was  followed  by  a 
simulation  forced  interannually  by  two  different 
atmospheric  products.  This  simulation  was  first  run  with 
ECMWF  atmospheric  forcing  for  the  period  1979-1995 
using  the  6-hourIy  ECMWF  reanalysis.  The  ECMWF 
product  is  widely  acknowledged  as  the  best  available  and 
is  available  over  an  interval  long  enough  to  include  a  wide 
range  of  variability.  In  particular,  it  extends  back  far 
enough  to  include  the  1982-83  El  Nino  (with  some  lead- 
time),  an  El  Nino  which  had  at  least  a  decadal  impact  on 
Pacific  Ocean  variability  (Jacobs  et.  al.,  1994).  1991 

this  simulation  was  continued  using  FNMOC  NOGAPS 
forcing  and  was  run  through  the  present.  In  all  of  the 
interannual  runs  the  temporal  mean  of  the  wind  forcing  is 
replaced  by  the  annual  mean  from  the  Hellerman  and 
Rosenstein  (1983)  (HR)  wind  stress  climatology  used  to 
force  the  climatologically  forced  simulation.  This 
minimizes  additional  spin-up  effects  when  the  interannual 
forcing  is  started  and  it  avoids  the  need  for  separate  spin- 
ups  for  different  wind  products.  Overall,  HR  wind  stress 
forcing  gives  the  most  realistic  model  means  of  the  wind 
products  we  have  tried. 
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The  interannual  simulations  were  used  to  assess  the 
model  realism  and  dynamics  in  representing 
atmospherically  forced  variability  on  a  wide  range  of  time 
and  space  scales  and  dynamical  regimes.  The  interannual 
simulations  are  critical  for  comparisons  to  data  sets  taken 
over  specific  time  frames.  They  also  help  us  investigate 
the  model’s  ability  to  produce  a  wide  range  of  oceanic 
variability.  In  addition,  interannual  simulations  attract 
greater  interest  from  researchers  outside  of  NRL.  The 
1/32°  model  resolution  is  an  unprecedented  opportunity  to 
study  regional  oceanography  and  dynamics  at  high 
resolution  in  the  context  of  the  global  ocean  circulation 
(which  is  also  influenced  by  the  high  resolution).  Major 
scientific  objectives  include  investigating  (a)  the 
interactions  between  the  thermohaline  component  of  the 
circulation,  the  wind-driven  component  and  the  mesoscale 
eddy  field;  (b)  the  influence  of  upper  ocean — topographic 
coupling  via  baroclinic  instability  on  the  preceding  and  on 
the  pathways  of  upper  ocean  currents;  (c)  the  influence  of 
the  global  ocean  circulation  on  regional  dynamics;  and  (d) 
Low  Latitude  Western  Boundary  Currents  (LLWBCs)  and 
their  roles  in  interbasin  exchange. 

The  interannual  simulation  has  been  followed  by  (1) 
ocean  nowcast  experiments  with  data  assimilation,  and  (2) 
ocean  forecast  experiments  initialized  from  (1)  later  in 
FY03.  All  of  the  data  assimilation  experiments  are 
initialized  from  the  FNMOC  NOGAPS  forced  simulations 
and  will  use  this  forcing  during  data  assimilation  and 
forecasting  experiments  except  for  forecasting 
experiments  where  the  atmospheric  forcing  is  relaxed 
towards  climatology  during  the  forecast.  This  will  be 
done  to  assess  the  impact  of  atmospheric  forcing  on 
oceanic  forecast  skill  as  a  function  of  time,  region  and 
dynamical  regime.  In  addition  to  comparison  with  the 
ECMWF-forced  simulations,  the  FNMOC  forced 
simulations  will  be  compared  with  data-assimilative  runs 
to  assess  the  impact  of  the  data  assimilation.  In  all  cases 
these  comparisons  will  be  done  in  the  context  of  model- 
data  comparisons  using  contemporaneous  data. 

The  data  assimilation  experiments  are  being  used  to 
improve  and  optimize  data  assimilation  techniques  at 
1/32°  over  a  wide  range  of  ocean  regions  and  dynamical 
regimes.  The  interannual  simulation  was  used  to  provide 
statistical  parameters  for  the  subsurface  statistical 
inference  technique  (Hurlburt  et.  al.,  1990)  used  to  project 
the  surface  data  downward  to  update  the  lower  layers  of 
the  ocean  model.  It  also  provides  a  high  resolution  global 
temporal  mean  SSH  which  must  be  added  to  the  SSH 
deviations  obtained  from  satellite  altimetry.  Surface 
dynamic  height  from  hydrographic  climatologies  is 
inadequate  for  this  purpose.  NLOM  is  designed  to  provide 
accurate  mean  sea  surface  height.  It  agrees  well  with  the 
hydrographic  climatologies  but  provides  a  much  sharper 
depiction  of  mean  currents.  Because  of  the  preceding  and 
in  addition,  the  importance  of  model  dynamical 


interpolation  skill  in  assimilating  ocean  data,  and  the 
model  skill  in  converting  atmospheric  forcing  into 
oceanic  information,  the  ocean  model  is  an  integral 
component  of  the  data  assimilation  scheme.  Model 
simulation  skill  is  also  essential  for  skillful  30-day 
forecasts.  The  forecast  experiments  will  be  used  to  assess 
model  forecast  skill  as  a  function  of  region  and  dynamical 
regime.  In  addition,  forecast  skill  will  be  used  as  one  of 
the  gauges  to  assess  data  assimilation  skill.  Thus, 
research,  development,  test  and  evaluation  running  of  the 
1/32°  global  model  with  and  without  data  assimilation  and 
running  it  in  forecast  mode  from  data-assimilative  states 
are  essential  for  transition  of  a  successful  1/32°  global 
NLOM  nowcast/forecast  system  to  operational  use  at 
NAVO.  Finally,  an  assimilation  experiments  run  from 
1993  (the  start  of  TOPEX/POSEIDON  altimeter  data)  up 
to  near  real  time  is  planned  to  assess  data  assimilation  and 
forecast  skill  over  a  wide  range  of  variability  in  each 
region.  This  assimilation  run  will  allow  researchers  to 
follow  the  evolution  of  observed  oceanic  anomalies  with 
unprecedented  resolution  and  with  space-time  continuity 
that  is  generally  not  possible  using  raw  data. 

3.  Results 

After  30  years  of  climatological  spinup,  the  ocean 
model  was  run  interannually  spanning  the  period  1979- 
present.  This  1/32°  interannual  simulation  shows 
increased  realism  and  accuracy  compared  to  its  1/16° 
counterpart  (run  under  an  earlier  DoD  Challenge  project) 
attributable  to  the  increase  in  horizontal  resolution.  An 
example  illustrating  the  value  added  of  finer  resolution  is 
shown  in  Figure  1 . 

Figure  1  shows  a  comparison  of  sea  surface  height 
(SSH)  variability  in  the  Gulf  Stream  region  from  the 
1/16°  and  1/32°  global  models  and  from  Topex/Poseidon 
altimeter  data  for  1979-1999.  The  pattern  and  amplitude 
of  the  variability  reveals  valuable  information  about  the 
models’  ability  to  simulate  specific  features  of  the  flow 
field.  In  the  1/32°  model  the  high  variability  extends 
farther  to  the  east  and  matches  the  pattern  and  amplitude 
from  Topex/Poseidon  much  better.  However,  the 
northeastward  extension  of  variability  east  of  47°W  is  too 
weak  in  both  models,  although  it  is  still  a  region  of  high 
model  variability  in  comparison  to  most  ocean  regions. 
Starting  from  west  to  east,  all  three  show  the  narrow  band 
of  veiy  high  variability  west  of  68°W  and  a  broadening 
east  of  that  longitude.  There  is  a  local  bulge  of  high 
variability  to  the  south  in  the  region  of  the  New  England 
Seamount  Chain,  but  the  1/32°  model  shows  better 
longitudinal  agreement  with  Topex/Poseidon.  Between 
45°W  and  50°W  the  1/32°  model  and  Topex/Poseidon 
show  a  corridor  of  high  variability  wrapping  around  the 
nonlinear  recirculation  gyre,  which  is  lacking  in  the  1/16° 
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model.  This  shows  that  the  1/32°  model  simulates  realistic 
eastward  penetration  for  the  nonlinear  recirculation  gyre. 


Figure  1.  Comparison  of  SSH  variabiiity  (1979- 
1999)  between  the  1/16°  (upper  panei)  and  1/32° 
giobai  models  (middle  panel)  and  analysis  of 
Topex/Poseidon  data  (lower  panel)  by  Jacobs 
(NRL).  The  5  lines  overlaid  on  each  SSH 
variability  plot  are  the  Gulf  Stream  mean  axis 
(center),  standard  deviation  about  the  mean  axis 
and  extreme  positions  determined  from 
Topex/Poseidon  data  (Lee  et  al.,  1997). 

Statistics  from  comparisons  (1997—1999)  of  modeled 
vs.  observed  SSH  for  104  tide  gauge  locations  throughout 
the  world  ocean  are  shown  in  Figure  2.  The  modeled  SSH 
is  able  to  generally  reproduce  the  observed  sea  level 
fluctuations  well,  with  a  median  correlation  of  .62.  The 
regions  of  highest  correlation  coincide  with  regions  in  the 
tropics  and  along  eastern  boundaries.  There  are  several 


regions,  however,  where  the  correlations  are  either  low  or 
negative.  Typically,  these  are  regions  where  the  SSH 
variability  is  a  nondeterministic  response  to  atmospheric 
forcing  due  to  flow  instabilities.  In  these  regions  ocean 
data  assimilation  helps  greatly  in  reproducing  specific 
features  of  the  observed  variability  such  as  individual 
current  meanders  and  eddies  (e.g.,  Hurlburt  et.  al,  2002). 
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Figure  2.  NLOM  vs  observed  monthly  sea  level 
correlation  (upper  panel)  and  RMS  difference  (lower 
panel)  statistics  at  104  locations  throughout  the  world 
ocean  from  the  years  1979-1999.  The  median 
correlation  is  .62  and  the  median  RMS  difference  is  7.2 
cm. 

Ocean  nowcast  experiments  with  data  assimilation 
initialized  from  the  interannually  forced  simulation  are 
currently  ongoing.  The  1979-present  interannual  model 
results  were  used  to  generate  statistics  and  a  model  SSH 
mean  for  use  in  these  data  assimilative  experiments.  The 
model  is  assimilating  SSH  from  3  satellite  altimeters 
(ERS-2,  GFO  and  JASON- 1)  and  sea  surface  temperature 
from  satellite  IR  imagery.  Assimilation  of  satellite 
altimeter  data  is  critical  in  allowing  the  model  to  map 
individual  current  meanders  and  eddies  (see  Figure  3). 


296 


ci=aiwc 


6.  CTA 


Figure  3.  Sea  surface  temperature  for  11  Feb 
2002  (upper  panel)  and  sea  surface  height  for  2 
May  2002  (lower  panel)  from  the  data 
assimilative  model.  Note  the  good  agreement 
between  modeled  features  and  operational 
satellite  IR  frontal  analyses  performed  by  NAVO 
(overlaid  black  and  white  lines).  The  black  line 
segments  are  based  on  IR  data  >  4  days  old. 

We  are  currently  testing,  evaluating  and  tuning  the 
data  assimilation  technique  used.  The  remainder  of  this 
fiscal  year  we  will  complete  the  ocean  nowcast 
experiments  with  data  assimilation  and  perform  ocean 
forecast  experiments.  The  completed  system  is  planned 
for  transition  to  NAVO  later  in  2003. 

4.  Significance  to  DoD 

Applications  for  this  nowcast/forecast  system  include 
assimilation  and  synthesis  of  global  satellite  surface  data; 
ocean  prediction;  optimum  track  ship  routing;  search  and 
rescue;  anti-submarine  warfare  and  surveillance;  tactical 
planning;  high  resolution  inputs  for  other  models  and 
shipboard  environmental  products;  environmental 
simulation  and  synthetic  environments;  observing  system 
simulations;  ocean  research;  pollution  and  tracer  tracking 
and  inputs  to  water  quality  assessment. 

5.  Systems  Used 

IBM  SMP  at  NAVO  MSRC 


Climate/Weather/Ocean  (CWO) 
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